DOCUMENT HESOME 



ED 132 J8j 

TITLE 

INSTITUTION 

SPONS AGENCY 
PUB DATE 
CCNIFxACT 
NOTE 



03 



CB 009 136 



EDHS PKICE 
DESCRIFTOES 



VocaticRcil Education Curriculuai Specialist (VECS) . 
Module 14: General Methods and Techniques of 
Education al Evaluation* Study Guide* 
(Teaching/Learning Module) - 

American Institutes for Research in the Behavioral 
Sciences, Palo Alto, Calif, 

Office of Education (DHEW) , Washington, D.C. 

76 

OEC-0-7a-9286 

57p,; tor related documents see CE 009 129-136 ar.d CE 
009 192-206 

MF-$0,&3 HC-$3-50 Plus Postage, 

*Curriculum Development; Curriculum Evaluation; 
^Evaluation Methods; Higher Education; Instructional 
Materials; Instructional Programs; learning Modules; 
♦Models; Post Secondary Education; *Program 
Evaluation; Secondary Education; Specialists; Study 
Guides; Teacher Education; *Vocational Education 

ABSTRACT 

One of 15 core modules in a 22-mod\" ^ series designed 
to train vocational education curriculum specially *r (VECS), this 
guide is intended for use by both instructor and student in a variety 
of education environments, including independent study, team 
teaching, seminars, and workshops, as well as in more conventional 
classroom settings. The guide has five major sections. Part I, 
Organization and Administration, contains an overview and rationale, 
educational goals and performance objectives, recommended learning 
materials, and suggested reference materials. Part II, Content and 
Study Activities, contains the content outline arranged by goals. 
Study activities for each goal and its corresponding objectives 
follow each section of the content outline- Content focus is on the 
purposes and components of the prominent types or models of 
educational evaluation and their applicability to the evaluation of 
vocational education programs. Part III, Group and Classroom 
Activities, suggests classroom or group activities and discussions 
keyed to specific content in the outline and to specific materials i^* 
the list of references Part IV, Student Self-Cieck, contains 
questions directly related to the goals and objectives of the module, 
which may Le used as a pretest cr posttest. Part V, Appendix, 
contains suggested responses to he study activities from part II and 
responses to the ntudent self-checks, (HD) 

♦ ♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦ #11 ♦*♦♦**♦*♦*♦♦ 

Documents acquired by ERIC ii^clude many informal unpublished * 
materials not available from .er sources, ERIC makes every effort 
to obtain the best copy availajj.e. Nevertheless, items of marginal 
reproducibility are often encountered and this affects the quality 
of the microfiche and hardcopy reproductions ERIC makes available 
via the ERIC Document Reproductjon Service (EDRS) o EDRS is not 
responsible for the quality of the original document. Reproductions 
supplied by EDRS are the best that can be made from the original. 



EKLC 



0 



ERIC 



t; . ,. J t-; 



^^^^ ^i^j ^ -^^ id w 



-^^^5^ r^-p^A f?^> ^ 



STUDY GUIDE 

(TEACHING/ LEARNING MODULE) 



ne■:;^; .^^:r ] 1 / i'^:fi'}ct the pcsUion 
(lOulri bo i nf errea. 



-Study Guide- 



Module 14 

GENERAL METHODS AND 
TEChAilQUES OF 
EDUCATIONAL EVALUATION 



4 



Uiis dvuuim'nt. is ufiu or a series of teach i nq ' 1 eai'fi i 'uj modules 
di^^siyned to itMin Vocational Lclucation Cudm cul uir, specialists. 
Ihe titU.-, of all individual ly avail<iLde docunienis in this series 
appear r-ei -w: 



INTRODUCTORY MOOULtS 

1. Tfie Scope or' Vocational Education 

2. Roles of Vocational Educators in Cui'riculuin Management 

3. Curretit Trends in Vocational Education 
Or^]a^ ization of Vocational Education 

5. l.o()i slat ive Mandates for Vocational Education 

6. The "^reparation of Vocational Educators 



(. ,:^!E r-*G0::LES 

i. Important Differences Among Learners 

Learfiing Processes and Outcomes 
3. Applying Knowledge of Learning Processes and Outcomes to 
Instruction 

Assessing Manpower Needs and Supply in Vocational Education 
5. Laying the Groundwork for Vocational Education Curriculum Design 
D. Selecting Instructional Strategies for Vocational Education 

7. Derivation and Specification of Instructional Objectives 

8. Development of Instructional Materials 

9. Testing Instructional Objectives 

10. Fiscal Management of Vocational Education Programs 

11. Introducing and Maintaining Innovation 

12. Managing Vocational Education Programs 
Basic Concepts in Educational Evaluation 

14. General Methods and Techniques of Educational Evaluation 

15. Procedures for Conducting Evaluations of Vocational Education 



SEMINARS AND FIELD EXPERIENCE MODULE 

(Seminars in Authority Roles and the Curriculum Specialist in Voca- 
tional Education, and Leadership Styles and Functions of the Curric- 
ulum Specialist in Vocational EJ-.cation; field work in Project Design 
and Administration, Operation of School Programs, Evaluation of 
School Programs, Educational Research and Development, and State, 
Regional, and Federal Program Supervision) 



INSTALLATION GUIDE 



For ale by the Superintendent of Documents 
U. S. Government Printing Office 

Washington, D.C. 20402 Price $ 

Stock No. 



PREFACE 



\u\\) IS a vu(:*lk)fK*i lhIn^.cU ioti curricul i.Mii ".poc i a 1 i s t ? The answer" 
:j l^iis i]'jestiofi f>ut as si^i^plo as it .iiiqht a■^w^^^ A vocational 
■-.Ki^.at 1'^:. Mir-r 1 Mjp specialist is likely to v^ot^k in nany differ^ent 
..ap'.iL i t les , i!u:li)d'ruj, b\ii not limited to: instructor, department 

1 i rpersor^ doan or voca t iorail - techm ca 1 education, vocational super^- 
,^snr\ ;'>riru jpal stable or local dir-ector of vocational education, and 
L 1 c J ] j^". i_aor*d i nc tor . 

Tml' specitiiir^t is, per'naps, more identifiable by his/her i^espon- 
sKMiiLies, ,vMi..h ificljde, but are not limited to: 

• p1annifK|. jr.janizing, actualizing, and controlling the work 
or an educational team performed to determine and achieve 
object i ves . 

0 planning, organizing, and evaluating content and learning 
processes into sequential activities ihat facilitate the 
achievement of objectives. 

• diaijnosinq present and p>^ojeL^.?d training needs of business, 
industry , educational i nsti *:uti ons , and the learner. 

• knowing, comparing, and analyzmg different theories of curric- 
uluin development, management, and evaluation and adapting them 
for use in vocational -technical education. 

This teaching/learning module is part of a set of materials repre- 
senting a comprehensive curriculum development project dealing with the 
training of vocational education curriculum specialists. The purpose 
of this two-year project was 1) to design, develop, and evaluate an^ 
advanced-level training program, with necessary instructional materials 
based on identified vocational education curriculum specialist compe- 
tencies, and 2) to create an installation guide to assist instructors 
and admirn'strators in the implementation process. 

The curriculum presented here is, above all else, designed for 
flexible installation. These materials are not meant to be used only 
in the manner of an ordinary textbook. The materials can be used 
effectively by both instructor and student in a variety of educa- 
tional environments, including independent study, team teaching, 
seminars, and workshops, as well as in more conventional classroom 
settings. 

Dr. James A. Dunn 
Principal Investigator and 
presently Director, 
Developmental Systems Group 
American Institutes for Research 
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PART I 

ORGANIZATION AND ADMINISTRATION 



Guidelines 

This study guide has five major sections. Each section contains useful 
ir^foniiat ion , sucgestions, and/or activities that as"^ist in the achievement 
of t[ie competencies of a Vocaiional Education Curriculum Specialist. Each 
fnajor section is brie'ly described below. 

^.T_ Ij^ 0RGANIZA[10N AND ADMINISTRATION 

PART I contains an Overview and Rationale, Educational Goals and Performance 
Objectives, Recommended Learning Materials, and Suggested Reference 
Materials. This section will help the user answer the following questions: 

• How is the fiKidule organized? 

• What is the educational purpose of the module? 

• What specifically should the user l^arn from this module? 

• What are the specific competencies emphasized in this module? 

• What learning materials are necessary? 

t What related reference materials would be helpful? 

PART II: CONTENT AND STUDY ACTIVITIES 

Part II contains the content outline arranged by goals. The outline is a 
synthesis of information from many sources related to the major topics 
(goals and objectives) of the module. Study activities for each goal and 
its ^^^rresponding objectives follow each section of the content outline, 

. g students to complete the exercises related to Goal 1 before going 

to .oal 2. 

rf\P.^ in^ GROUP AND CLASSROOM ACTIVITIES 

ne "Activities-Resources" column in the content outline contains refer- 
ences to classroom or group activities and discussion questions related to 
specific content in the outline. These activities ard discussion questions 

JO 



are located in PART lil and are for optional use of ei tfier the instructor 
or the student. Both the classroom activities and discussion qt^estions are 
accompanied by sucjnested responses for use as fielpful examples only-- they 
do not represent conclusive answers to the problems and issues addressed. 
Also contained in the "Activities-Resources" column are the reference 
numbers of the resources used to develop the content outline. These 
reference numbers correspond to the numbers of the Suggested Reference 
Materials in PART L 

PART IV: STUDENT SELF-CHECK 

PART r.' contains questions directly related to the goals and objectives of 
the module. The self-check may be used as a pre-test or as a post-test, 
or as a periodic self-check for students in determining their own progress 
throughout the module. 

PART V: APPENDICES 

Appendix A contains responses to the Study Activities from PART II, and 
A[)pendix B contains responses to the Student Self-Check. The responses 
provide immediate feedback to the user and allow the module tg be used 
more effectively for individualized study. They have been included in the 
last part of the module as appendices to facilitate their removal should 
the user wish lu jse them at a later time rather than concurrently with 
the rest of the modul e. 

Approximately 30 hours of outo^-class study will be necessary to complete 
this module. 

Overview and Ratio nale 

Today's educational evaluator has a variety of evaluation theories that 
have been operational i zed in the form of evaluation models from which 
to choose. In this module, the more prominent types of educational 
evaluation models are examined in order to provide the student with a 
working knowledge of the skills and techniques that may be required of 
an educational evaluator. 



This module is divided ii^Lo five iiiajor content Tea:.. The first area 
delineates the purposes evaluation as they ar^ most conimonly con- 
ceived. Each of the last fnur content areas is an examination of one 
of the four "types'* of educational evaluation. 'Whenever possible, 
seminal papers upon whi'ch particular evaluation types have been based 
are included as part of the learning activities. Suggested classroom 
discussions concern the applicability of each evaluation type to voca- 
tional education, and specific examples froi.. vocational education are 
found in the learning exercises. 

All readings suggested in this module should be studied with respect to the 
purposes of evaluation, the roles of the evaluator, and the steps to be 
taken in conducting an educational evaluation. Four general models of 
educational evaluation are presented. These four models were introduced 
in Module 13: Basic Concepts in Educational Evaluation and are labelled 
goal-attainment evaluation, judgmental evaluation emphasizing intrinsic 
criteria, judgmental evaluation emphasizing extrinsic criteria, and 
decision-facil i tat ion evaluation. 

Goal -attainment evaluation focuses essentially on the extent to which 
program goals are translated into behavioral objectives and achieved by 
program participants. . 

Judgmental evaluations in general are based on assessments of progr-.;!! 
components by the evaluator or expert. 

Judgmental evaluation emphasizing intrinsic criteria addi'esses the prac- 
tice of accreditation or certification of educational programs with re- 
spect to facilities, size of t^e library, pupil-teacher ratios, degrees 
earned by program staff, and other standards related to program compo- 
nents . 

Judgmental evaluation emphasizing criteria extrinsic to the program 
attends to criteria relating to the description and judgment of fac':ors 
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atiori'i, and r(:\\j]l\, [)l^tIn^t .r, .irf dr.ivji t^^- t.wfCD tjM' roh- of '"mI- 
iJcJtion in progrdiii d<j v<' 1 fJpfiK-ri * < rly iinplf'o-nt.dt. ion ( f uriii.i t i v«' ; ond 

tho Jij(.]fjiii(*n t. fjf Ih^' T'f f^'Ctj v^.'! i tfn- t.ol.il pro^jr^in {'Aiwdiuil \ ye , , 

U^m: i 'i i on- t (iC i 1 I ttJ U (HI v.i 1 u.) t i fjn furt.hor flofirj<", tJu- o vd 1 u.i t.{jr * '. roN- 
v/ith rir,\U!(,l to Uu) stfjfjf:'> of (nofjrMiit dr'v<' 1 oj^iMcr, * , from p I .triri i ri^j t.hrou'jfi 
i nip 1 f.Min'n tfj I i fJM t.rj firuil oijtj oc^". . At. ♦•.kJi tjf tho f or fiui 1. 1 v - '.uiia i<i r i vo 

con t i rujuiM , r'olo (jf tfio o\ .it.or i to providf i n f o r ri.i 1. 1 on to pro'jf'^ff 

<.lovo 1 opor"i , iiwifWJ <jor', , i mp 1 onuTit..! lju", , ',pf>n'iOr '> , <)no (jtfMT" do{. : ujri -I'l.ih'r . 
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Goal end Objectives 



:f\"r. m;-' '-'fi'ifr, ^ii i i;'*^ Af lori/^i fvAiuATiofj Ann Bi. awapc fjf 

Min;^ Al'l'l If.AfM! 1 ! ^ M) :jf iVAlDAIIOfl Of VOCA 1 1 (JfJAI. LWh 
f A I I fj'; ['!^f)f,(^AM', . 

fi(>ipMi7^' M.ll '.t.it..* t'lf ivr'riill pur'iiosc f)f rMf;h of l\u' 
f our r «i jf)r f'V.i I u.it. i on lypf"", . 

f )i) |('( t I v^' M.I/' Dffiru' f.hf roW* of t.fio r»v<i lu«it or' in (Mcfi 
iiioflf'l f)r lypo of ov«i liJ«i(. if)n. 

f 'I, f i Vf MJ< t.fif' r)pcriit. ion<jl slops rofjuirfd in fX)n- 

I nfj <ui fduf <jt. i on«j I ov<i I \it\t i on . 

Ot))0(tivf' M.M (ifv^'lop .irul list orilori.j for voCriUonal 

(♦flij( .!!. ion I'va I U(jtJ fjri for fNJ(,fi of 0)0 fiiajor 
fV'i I ij<)l i oil t.ypt'S , 
)f( t 1 v<* I^.IS I) i st, i rifju i sfi .Jiiioritj .jctJ v i t.ios lU^il ^iro cfuir'- 
,H tfr t. 1 «is fjfj.i I -«it.t.nni(K»nt ovalucjtion, 
jijtlfjiiiont.n I ov.ilu.iUon oiii[)fi,js i / i ruj Intrinsic 
f.r it<'ri.i, .juHf)iiK»r)t,ci I rwaluation (Miiphas i / i nfj 
jvtrirr.if. fritoria, anrl rlorjs ion-faci li- 
t -1 ! \iir\ f'V.i I uat Ion . 
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Recommended Materials 



1. Ander-.on, Scarvia B., et al , "Evaluation Concepts" in Encyclopedia 

of _E_dm:ational Eva'luation. San Francisco: Jossey-Bass, Inc., 

1973: 

2. Cronbach, Lee, "Course Improvement through Evaluation." Te acher s 

Coljecje Record, 64 (1 963): 672-683. 

3. Tyler, Ralph W. "General Statement on Evaluation." Journ al of 

Educational Research, 35 (1942): 492-501. 

4. Po()haiii, W. James. Excerpt from Chapter 2, "Contemporary Conceptions 

of Educational Evaluation" in Educational Evalu ation . Englewood 
Cliffs, New Jersey: Prentice-Hafl" ,' IncV, T975', pp. 24-25. 

5. SLikfs Robert E. "The Countenance of Educational Eval uation . 

Teachers Col l ege_Recor_d, 68 (1967): 523-540. 

6. Alkin, Marvin C, and Fitz-Gibbon, Carol T. "Methods and Theories 

of Evaluating Programs." JojjrnaJ_o^ jleseaj;^^ ^ and Development 
in Education, 8 (1 975): 2-15."' ' 

NOTE: The above materials are necessary if students are to complete 
ade(]Ucitely tfie study activities following the content outline. 



Suggested References 

L Alkin, Marvin C. et al . Evaluation and J)ed The Title 

vil Experience. Los AngeTesT iJni versTty of "CaTTforni a , Center 
for t.he S'tucly of Eva I u.j tion , 1974. 



ririfl rit/-Gibbon, Carol T. "Methods and Theories of Evalu- 
at/inq Programs." Journal of Research and Development in Educa- 
t.ion, 8 (Spring 1975) :" " ^"-TiT." 

3. CnMiiin, Lawrence; The Transformation of _th^^^^ New York: 

Vinl.age liooks , 1961'. 

4. fiha/rilah, I. A. "Iho Role of Vocational Education in Improving 

Skills and Eorning Capacity in the State of Ohio: A Cost- 
nruiefiL Study." Athens, Ohio: College of Business Adminis- 
Lral.ion, Otiio University, November 1972. 
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Multiple Criterion Measures for the Evaluation of the Effective- 
ness of School Programs." Educational an d Psycho lo gical Mea- 
surement, 27 (1967): 931-943. 
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cion. Today a nd Tomorrow , edited by Gerald G. Somers and J. 
Kenrietti Little. Mddison, Wisconsin: The University of Wis- 
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9. National Society for the Study of Education. The Curriculum: Retro - 
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Un^versi^y of Chicago Press, 1971. 



10. National Society for tho 3tudy of Education. Educational Evalua - 
tion: New Roles, New Means . The Sixty-eighth Yearbook, 
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11. National Study of Secondary School Evaluation. Evaluative Cri- 
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Colui .>i.3, O^hioT* Charles' E'.' MerriirPublishing Company, 1974. 
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I 7 



-8- 

EKLC 



Part II: 



Content and Study Activities 



J 8 



PART II 

CONTENT AND STUDY ACTIVITIES 



God 14.1 



ERIC 



Content Outline 



Goal 14J: Be Aware of the Purposes and w: 
Components of the Prominent Types or 
Models of Educational Evaluation and Be 
Aware of Their Applicability to the 
Evaluation of Vocational Education 
Programs. 

y/////////////////// /////////////////////y 





Activities-Resources 



Purposes of Evaluation 

1. Evaluation in education has been and is used 
for many purposes. Among these are: 

a. classification of students; 

b. justification of expenditures; 

c. confirmation or rejection of hypotheses 
regarding teaching methods, instructional 
materidls , etc. ; 

d. public relations; 
0. accountabi li ty ; 

f. provision of guidance information; 

g. course improvement; and 

h. improvement of teachers, administrators, 
and other personnel (10), (12). * 

2 . Ev aluation for Course Improvement 
a. Prior to Ralph Tyler's writing in the early 

1940s, theorists and writers in the educa- 
tional testing movenent spoke only vaguely 

I J) 
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(10) Educational Evalua - 
tion : New Roles . 
New Means , pp. 
10-n. 

(12) Educational 

Evaluation . Chap. 
2. pp. 20-44. 

* See the Glossary of 
Terms at the begin- 
ning of Part III. 



Content Outline (continued) 



about how what was then considered evalu- 
ation could be used to improve education 
and to demonstrate the merit of existing 
programs (3). 
b. Lee J. Cronbach, in his essay, "Course 
Improvement through Evaluation," took 
issue with the prevailing purposes of 
evaluation and asserted that the improve- 
ment of curriculum could not be brought 
about by using the technology of the day. 

B. Goal "Attainment E valuation 

1. Models based on goal attainment conceive of 
evaluation primarily as the determination of 
the degree to which a program's or project's 
goals are achieved. 

2. Ralph Tyler's model was first formally used in 
the Eight-Year Study and explained in a 1942 
paper. Widely disseminated in Basic Principles 



(3) The^ Transformation 
of the School , 
pp. 252-253. 



of Curri c ul urn arid Instructi on , the model has 
four steps. 

a. Formulate educational goals according to an 
analysis of three goal sources (students, 
society, and subject matter) and two goal 
screens (psychology and philosophy). 
Transform goals into behavioral objectives. 
Conduct program. 

Measure student progress to assess the 
degree to which objectives were met (3), 
(19). 

Robert: L. Hammond also conceives of evaluation 
in terms of whether or not objectives are met. 

-10- 



b. 
c. 
cl. 



(19) Basic Principles 
of Curricul um and 
Instruction . 



Content Outline (continued) 



To determine this, he identifies five steps 
that should be taken. 

a. Isolate aspects of the program to be 
evaluated. 

b. Define the relevant institutional and 
1 nstructional vari ables . 

c. Specify objectives in behavioral terms. 

d. Assess the behavior described in the 
objectives . 

e. Analyze goal attainment results (5). 
4. Metfessel and Michael expand the goal- 
attainment model by attempting to establish 
different classes of criterion measures that 
can by used to reflect goal attainment. There 
are eight steps in the Metfessel -Michael 
approach to evaluation. 

a. Involve members of the total community. 
Construct broad goals but specific 
objectives . 

Translate objectives into communicable and 
usable form. 

d. Develop measurement instruments to assess 
accomplishment of objectives. 

e. Conduct periodic measurements. 

f. Analyze data from measurements. 

g. Interpret analyzed ^.jta. 

h. Formulate recommendations for change or 
modification (7).* * 



b. 



c. 



(5) "Evaluation at 
the Local Level 
pp. 157-169. 

(7) "A Paradigm 

Involving Multiple 
Criterion Measures 
for the Evaluation 
of the Effective- 
ness of School 
Programs," pp. 
931-943. 

See the goal- 
attainment exercise 
(a) of Classroom 
Activity 1 in Part III, 

See Discussion 
Questions A and B 
in Part III. 



21 

-11- 



EKLC 



Content Outline (continued) 



C. Judgmental Evaluation Based on Instrinsic 




Cri teri a 




1. The United States has no federal ministry of 




education or other centralized authority 




exercising a single, national control over 




educational i nsti tutions . 




To ensure a basic level of quality, the prac- 




tice of accreditation arose in the iMted 





States as a means of conducting nongovernmental, 
peer evaluation of educational institutions anJ 
programs. Private educational associations of 
regional or national scope have adopted 
criteria reflecting the qualities or sound 
educational programs, a.id developed procedures 
for evaluating institutions or programs to 
determine whether or not they are operating at 
these basic levels of quality. 
2, Functions of accreditation are: 

a. certifying that an institution has met 
established standards; 

b. assisting prospective students to identify 
acceptable institutions; 

c. assisting institutions in determining the 
acceptability of transfer credits. 

d. helping to identify acceptable institutions 
and programs for the investment of public 
and private funds; 

e. protecting an institution against harmful 
Internal and external pressures; 

f. creating goals for the self-improvement of 
weaker programs and stimulating a general 
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raising of standards among educational 
institutions; 

g. involving the faculty and staff comprehen- 
sively in institutional evaluation and 
planning; 

h. establishing criteria for professional 
certification and licensing, and for up- 
grading courses offering such preparation; 
and 

i. providing one basis for determining 
eligibility for federal assistance (11). 

The procedure for accrediting institutions 
involves the following steps: 

a. Standards : The accrediting agency (which 
is composed of members from the institution^ 
it accredits), in collaboration with edu- 
cational institutions, establishes 
standards . 

b. Sel f-study : The institution or program 
seeking accreditation prepares a self- 
evaluation study that measures its perfor- 
mance against the standards established by 
the accrediting agency, 

c. Onsite Evaluation : A team selected by the 
accrediting agency visits the institution 
or program to determine firsthand if the 
applicant meets the established standards. 

d. P ublication : When it is satisfied that 
the applicant meets its standards, the 
accrediting agency lists the institution or 
program in an official publication with 
other similarly accredited institutions or 
programs . 
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(11) Evaluative 
Criteria. 



Content Outline (continued) 



e. Reeval uation : The accrediting agency 
periodically reevaluates the institutions 
or programs that it lists to ascertain 
that continuation of the accredited status 
is warranted. 
4. Evaluation based on intrinsic criteria is 

':onerally conceived by theorists as evaluation 
by checklist and professional judgment. Eval- 
uation may be by criteria sucli as: 

a. type of equipment in shops; 

b. facil ities; 

c. number and type of books in the library; 

d. 'qualifications of faculty; 

e. financial support (12).* 



(12) Educational 
Eval uation , 
pp. 24-25. 



D . Judgmental Eval uation Based oji Extrinsic Criteria 

1. The two most prominent extrinsic criteria eval- 
uation models have been proposed by Michael 
Scriven and Robert Stake. Scriven first drew 
evaluetors' attention to the formative- 
summative distinction. 

a. Scriven conceives of evaluation (including 
evaluation of the goals themselves) as 
ultimately an assessment of merit (15). 

b. Scriven calls his approach, which focuses 
on extrinsic criteria, "pay-off" evaluation 
The effects of the program on the students, 
teachers, administrators, and other rela 

vant groups. 

c. He advocates comparative evaluation because 
decision-making often invokes choices 
among competing options, thereby requiring 
a comparison of the competitors. 



* See Discussion 
Questions C, D 
and E in Part III . 
Also see the 
judgmental evaluation 
based on intrinsic 
criteria exercise (b) 
of Classroom Activity 
1. 



(15) "The Methodology 
of Evaluation," 
pp. 39-83. 
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d. He also proposes goal-free evaluation to 
remove any bias created by knowing the 
goals while trying to determine their 
effects (16). 
Stake's Countenance Model emphasizes descrip- 
tion and judgment di> the two chief operations 
of evaluation. 

a. He advocates that evaluators help to write 
behavioral objectives. 

b. He indicates that descriptive and 
judgmental acts are distinguished according 
to three phases of an educational program: 

(1) antecedent- -or conditions existing 
prior to instruction; 

(2) transaction- -a succession of engage- 
ment? that constitute the instruc- 
tional process; and 

(3) outcomes --or the effects of a program. 

c. Descriptive acts are divided according to 
what v;as intended and what was actually 
observed. 

d. Judgmental acts are divided according to 
whether they refer to the standards or 
criteria used in reaching judgments or to 
the actual judgments themselves. 

e. The Countenance Model continuously involves 
the evaluator in description and judgment-- 
from the beginning to the end of the 
program. 

f. Absolute vs. relative (comparative vs. 
opinion or personal judgment) criteria 
are used in judging.* 

2^. 



(16) "Pros and Cons 
about Goal -Free 
Evaluations," 
pp. 1-4. 



See Discussion 
Questions F and G 
and also the 
judgmental evalu- 
ation based on 
extrinsic criteria 
exercise (c) of 
Classroom Activity 
1 in Part III. 
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Deci sion-Faci 1 i tation Eval uations 
1 



The objective of these models is to provide 
educational decision-makers with the infor- 
mation they ficed to improve programs. 
Daniel Stufflebeam and the CIPP Model view 
evaluation as a continuing and cyclical process 
(17) (18) (12). 

a. Four types of decisions are served by eval 
uati on : 

(1) Context --Eval uation provides a 
rationale for the determination of 
objectives . 

(2) Input- -Evaluation provides information 
on how to employ resources so as to 
achieve program objectives. 
Process--Eval uation identifies defects 



(17) Educ ational Eval 

(la) 



uation and Deci- 



(3) 
(4) 



in procedural design. 
Product -'Evaluation measures and 
interprets the results of an instruc- 
tional program, 
b. Three steps are involved in each of the 
evaluations : 

(1) Del i neatinq ' information^ -The focus is 
put on information required by the 
deci sion-makers . 

(2) Obtaining- "The data in (1) above is 
collected and analyzed. 

(3) Providing -^The material of (2) above 
is put into a form useful to the 
decision-maker. 

Malcom Provus ' decision-facilitation evaluation 
model involves the comparison of performance 



(12) 



SI on Maki ng . 
"Fvaluation as 
Enl i ghtenment 
for Decision- 
Making." 

Educ ationa I Eval- 
uation ^ pp. 
33^37. 
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with standards in order to determine whether 
to iirprove, maintain, or terminate a program. 
There are five stages to the Discrepancy Model 
(14) (10): 

a. Design --documcnts the nature of the 
program incluc: uj objectives, resources 
required, and instructional activities 
needed. 

b. Installation s-sees whether or r^t tne pro- 
gram is congruent with its inscanation 
plans. Choices then are to terminate, 
proceed, alter performance, or alter 
standards . 

c. Process — is similar to Scriven*s formative 
evaluation, the CIPP process evaluation, 
and the CSE proqr^^.s evaluation (15) (17) 
(2) (6). 

d. Product - s siviar to the product and 
outcome stage, of the CIPP and CSE models 
respectively. 

e. Program Compari son --ana1yzes the cost- 
benefit of other competing programs.* 

The CSE (Center for the Study of Evaluation- 
UCLA) decision-facilitation model is composed 
of three stages and six substages (1) (2) (6) 
(12): 

a. Pre-Formative Evaluation Stage 

(1) Needs Assessment —pinpoint educationa 
needs with the purpose of identifying 
educati^. goals. 

(2) Program P1anning- -provides information 
regarding the types of programs that 
would meet the needs identified in 
a (1). 
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(14) D i scj;;^^pa ncy 
Evaluation . 

10) Educational 



(15) "The Methodology 

of Evaluation. " 
[17) Educational 
Evaluation and 
Deci sion-Making . 
(2) "Methods and 

Theories of Eval 
uating Programs.' 
(6) "The Center's 
Changing Evalu- 
ation Model ." 



Evaluation : 
New Roles , New 
Means , pp. 
"242-283. 



See Discussion 
Question H in Part 
III. 

[1 ) Evaluation and 
Decision Making , 
Chap. 1. 
(12) Educational 

Evaluation , pp. 
37-39. 



Content Outline (continued) 



Formative Evaluation Stage 

(1) ImpTementati on "provides information 
on the degree to which the program is 
being implemented as planned. The 
focus is on modification (2). 

(2) ProgresS '-provides information on 
student progress with a focus on 
learning the results of implementation 

Summative Evaluation Stage 

(1) Documentatio n-- provides information 
on the results of the program. 

(2) Outcomes -provides information or 
recommendations on whether to 
continue or discontinue the program.* 



(2) "Methods and 

Theories of Eval 
uating Programs: 



* See the decision- 
facilitation evalu- 
ation exercise (d) 
of Classroom 
Activity 1 . Also 
see Discussion 
Questions I, J and 
K. 
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study Activities 

IxiCis^d oyi yoiiv vc~::diK<j of thr content outline and ayiij additional 
vefeve-yiaes as suggested^ corvlete the following activities. 

Is Cronbach's definition of evaluation compatible with the one given 
in Module 13? With that conceptualized by Anderson in the first 
reading? 

List the three types of decisions for which Cronbach feels evaluation 
can be used. 

Why did Cronbach emphasize evaluation for course improvement over the 
other two decision situations? 

Although Scriven hadn't made the formati ve/summati ve distinction at 
the time of Cronbach's essay, Cronbach treated the point. At what 
point in a program did Cronbach advocate evaluation? Why? 

What role should comparison evaluation play according to Cronbach? 

How much effect did Cronbach believe follow-up studies had on course 
improvement? Why? 

Tyler states that evaluation in the schools chiefly serves the 
purpose of grading students, selecting students, and reporting to 
parents and school boards. He argues that comprehensive evaluation 
should be more than this. List the other purposes Tyler felt evalua- 
tion should serve. 

What current (but often hotly debated) educational phenomenon is the 
result of Tyler's first assumption underlying the evaluation of the 
outcomes of general education? 

What does Tyler propose as the basis of educational objectives? 
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10. Tyler recommends a definite procedure for the conduct of educational 
evaluation. All the steps of the process (except the last) appear 
to be oriented toward one specific goal. What is that goal? 

11. State the difference between intrinsic criteria and extrinsic criteria. 

12. Can judgmental models be properly characterized as systematic 
evaluation as that has been defined in Module 13 and characterized 
in the first reading? Why? 

13. What is the most common type of judgmental evaluation using 
intrinsic criteria? Who conducts that evaluation? 

14. List several criteria that an accrediting agency might employ in 
evaluating a vocational education program. 

15. Stake's approach to evaluation emphasizes two main operations of 
the evaluator. What are those operations? 

16. According to Stake, there are three phases of educational programs 
with which the evaluator must be concerned. List those three phases. 

17. What purpose does the rationale statement have in an educational 
evaluation? 

18. Unlike the goal -attai nment or accreditation models, the Countenance 
Model contains an elaboration on the manner in which judgments are 
made by evaluators in terms of comparisons. List the two types of 
comparisons Stake delineated, and give an example of each. 

19. Does Stake call for the evaluator to pass a personal judgment on a 
program? If not, who is expected to make the judgment? 

20. List several criteria that an evaluator using Stake's model might 
employ if evaluating a vocational education program. 
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21. The CSE model is a decision-facilitation evaluation model that has 
three distinct stages. List those three stages and the two sub- 
stages of each. 

22. Give an example of the type of evaluation that would be provided to 
decision-makers at ^^^ch of the six substages of the CSE model. 

23. The procedure involved with the CSE model might be considered to 
have at least 12 steps since each of the three stages has fo'jr 
evaluator activities. What are those four activities? 



24. What position does Alkin take regarding the evaluator's making 
judgments? 

25. What are the chief differences between formative and summative 
evaluation as Alkin defines them? 



26. List several criteria that might be used in a decision-facilitation 
evaluation of vocational education. 

Indicate what type of evaluation is characterized in each of the following 
situations* 



27. An evaluation team has been sent to review a Regional 

Occupational Center. The team visits the center site 
and spends five days talking to staff and students. 
The team's questions and observations focus primarily 
on the plans that have been devised for the center 
and the equipment and facilities that are available 
for carrying out these plans. 
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A group of evaluators assigned to evaluate a federally 
funded career education program feel that the objectives 
of the program are reproachable but tha+-. their mission 
is to provide information that will enable the program 
staff to achieve those goals. Because of their con- 
ceived mission, the evaluation group never expresses 
their disapproval of the basic direction of the project. 

During a series of workshops, a group of printing 
instructors develops a clear, defensible set of 
behavioral objectives for their programs in offset 
printing. After incorporating the objectives^ into the 
programs, they want to evaluate the programs in terms 
of the degree to which those objectives were achieved. 

An evaluation team assigned by the state department of 
education to evaluate several controversial but well- 
funded experimental vocational programs employing 
the "Cluster Concept" decides to base their evaluation 
on how many of the students are placed in jobs, 
where they are placed, and how well they do their 
jobs after placement. 

The vocational nursing department of a local 
community college has developed a number of 
goals for each course in their program; 
the department evaluates the success of each 
course on the basis of measurement data indicat- 
ing whether or not those goals have been 
satisfactorily achieved. 
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32. A nationally known evaluation center has contracted 

to supply evaluation assistance for a newly 
funded curriculum development project that is 
designed to prepare vocational curriculum 
specialists. The project staff, all curriculum 
specialists, want the evaluators to gather 
information that will heip them improve the 
instructional materials they are developing. 
The evaluation team assembles a variety of data 
for the project staff, including field tests of 
instructional materials, student reactions, and 
the reactions of practicing vocational educators. 

Wrapup Acti vi ty 

NOTE: To meet the basic requirements of this module, select one of the 
following activities and complete it as directed. If you wish to gain 
additional credit beyond the basic requirements, you may choose a second 
activity to complete. Consult with your instructor first if you wish 
additional credit. 

1. Select a high school, community college, or accredited private 
vocational school near you and ask to examine their latest 
accreditation report. In a four- to five-page report, detail the 
criteria that the accreditation agency used to evaluate the program 
or school, and give an example of a data collection method or 
technique that was used foi each of the criteria. 

2. Using the resources found in the References section, research 

one of the four types of evaluation models. In a four- to five-page 
paper defend the use of that model, or type of evaluation, as a 
viable method of vocational education evaluation. 



Ethics is an issue that is skirted in the evaluation literature. 
Write a paper detailing the ethical qualities you (and the experts) 
feel evaluators should have. Some questions to prompt your 
thinking might be: How much business will an evaluator get in 
the future if he consistently gives poor evaluations? Should 
evaluators accept evaluation contracts after programs have already 
been started? Does a decision-facilitation evaluator have a 
responsibility to tell decision-makers that the objectives of their 
program are reprehensible? 

Research economic or cost-benefit evaluation as a viable type of 
evaluation for vocational education. Is the cost of conducting 
such evaluations prohibitive? What if a program costs a lot and 
students don't find employment, but nevertheless they rate the 
program as beneficial tiiem in noneconomic ways? Do you think 
cost-benefit evaluations might justify large expenditures for 
vocational education? 

■'c Appendix A for possible ansvers.) 
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PART III 



GROUP AND CLASSROOM ACTIVITIES 
Classroom' Activities 



GLOSSARY 

The following terms are defined in order to clarify their meanings in 
the context of this module. 



CIPP . An acronym formed from the first letters of the four basic kinds 
of evaluation under the decision-facilitation model as advocated by 
Stuff lebeam. The letters represent context, input, process, and 
product. 

COMPARATIVE EVALUATION . Evaluation in which one process or product is 
compared with another or with others in terms of effectiveness, cost, 
acceptance, etc. 

CRITERION . A standard for judging and validating. 

EDUCATIONA L PHENOMENA . Observable objects, facts, events, or processes 
in the educational setting that are evaluated. 

FORMATIVE EVALUATION . Evaluation that is concerned with program 
improvement, and that generally requires evaluator intervention in 
the program or process. 

OBJECTIVE . A point of accomplishment that can be verified within a 
given time and under specifiable conditions, which, if attained, 
reflects progress toward achievement of a corresponding goal. 
Objectives are sometimes known as behavioral objectives, performance 
objectives, and terminal objectives. 

PR E-TEST and POST-TEST . A component of an evaluation system or research 
design that requires testing before and after instruction or implemen- 
tation of a program to determine the extent of student progress or 
achievement. 

RELIABILITY . A term usually applied to measurement instruments, 
indicatit.g that the instrument is consistent in the way it measures 
qualities or characteristics. 

36 



-25- 



SUMMATIVE EVALUATION . Evaluation concerned with determining overall 
program effectiveness; it generally requires little or no evaluator 
intervention in the program or process. 

SYSTEMATIC EVALUATION . A systematic, formal process of identifying and 
collecting information on educational phenomena to assist decision- 
makers in choosing among available decision options. 

VALIDITY . A term usually applied to measurement instruments, indi- 
cating that the instruments adequately cover what they are designed 
to cover, that they correlate with factors or traits they are 
designed to measure, and that they correlate with other measures 
of the same trait. 



NOTE: The following activity is designed for use in the classrcom to 
stimulate discussion on specific topics covered in this module. The ac- 
tivity is designed to be used following student self-study; however, de- 
pending on the background and abilities of your students, these activi- 
ties may not require previous study. 



1. Divide the class into several groups and allow them to brainstorm possi- 
ble solutions to the problems presented in the following situation. This 
same activity should be performed for each of the four evaluation types 
covered in this module. In other words, the same problems will be 
"solved" four times, once for each type of evaluation. This will allow 
the students to compare and contrast the merits and techniques of the 
various models. 



The Si tuation . Assume that you are an agency that has contracted to 
evaluate a federally funded experimental program in vocational education. 
Fourteen of the fifty high schools in a large urban school district are 
to implement the program, which is an experiment with the "Cluster Con- 
cept" in vocational education, but none of the schools have been selected 
yet. Schools to be included in the evaluation will be selected by the 
evaluation agency. The program will be funded for $1 million the first 
year, $]h million the second, and $2 million the third. Since the usual 
^Q7o of funding is earmarked for evaluation, money is no problem in con- 
ducting the evaluation. The operators of the program and the federal 
sponsors have given the evaluation team free access to the program and 
the sites, once they are selected. 
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The Problem. As a class, select one type of evaluation to conduct: 

(a) goal-attaininent, (b) judgmental -i ntri ns i c criteria, (c) judqmental- 

extrinsic criteria, (d) or decision-facilitation. Then break up into 
groups and: 

a. develop an evaluation plan (stick to the model chosen); 

b. select criteria for the evaluation; 

c. delineate methods that might be used to collect 
decision information; and 

d. construct a mock evaluation report on the chalkboard 
or easel (outline U) that includes the results and 
recommendations of the group; these, of course, should 
be consistent with the evaluation type. 



Activities for Addi tional Credit 

NOTE: These activities are designed for the student who wishes to 
obtain credit beyond the basic requirements of the module. You may 
choose to have the student write a paper on one of these activities, 
or discuss the activity with you, or you may select some other method 
for the student to complete the activity. 

1, Determine what role evaluation plays, or is expected to play, 
in maintaining quality vocational education programs. 

2, Examine the criteria defined by Wenrich and Wenrich (20) and 
determine which evaluation model they were chosen to serve. 
(Those criteria are included on pp. 267 to 269 of 
Suggested References #20.) 

3, Collect at least three instruments such as tests, question- 
naires, checklists, interview forms, etc., that have been 
used in evaluations and prepare ^ content (analysis of them. 
For what types of decisions might they supply information? 
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Many evaluation models stress the importance of assigning stu- 
dents randomly to control or experimental groups. Discuss the 
problems this might cause a vocational education administrator. 

IQ tests are being banned from use throughout the United States 
except in special circumstances. What are the implications of 
these test bans with respect to the problem of verifying that 
the abilities of students are equal in two or more groups being 
evaluated? 
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You have been asked by your district vocational education super- 
visor to describe, as briefly as possible, the four evaluation 
models presented in this module. On the basis of the information 
found in this guide fill v^ the comparison matrix below, 
or construct your own iiia.i^ix. 



ATTAINMENT 


JUDG'.ENTAL-- 
NTRINSIC CRITERIA 


JUDGMENTAL-- 
EXTRINSIC CRITERIA 


DECISION- 
FACILITATION 


























• 








) 

1 

I 








) 
I 
) 
) 

1 

1 
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Discussio n Questions 

A. What are some limitations of goal -attainment evaluation in light of 
the fact that contemporary thought leans toward evaluation for 
purposes of improvement? 

(The value of the goals themselves is not a concern, but only whether 
they are met or not. Goal -attai nment evaluation is after-the-fact.) 

B. The last two paragraphs of the Tyler reading selection contain a 
discussion of the uses of evaluation. How often are those same points 
brought up in "modern" evaluation theory? 

(Tvler^ main points were that evaluation is a recurring process 
ivoWi'.g the formulation of objectives, the classification of their 
defini:ion. the study of students' reactions, and a continuous effort 
to Interpret students' reactions in order to help improve the education 
of individual students. 

Evaluation was defined as a continuous, cyclical process which would 
be used for the continued Improvement of the program of education. 
Although tlie process of evaluation is generally conceptualized much 
differently today, the general purposes remain the same.; 

C. Although accreditation evaluations are generally not "pushed" anynwre. 
what benefits to staff and students might accrue from conducting the 
type of intensive solf-study required by an accreditation evaluation? 

(Critical self-analysis might lead to enlightenment and avoid 
stagnati on . ) 

D. What roles do contont specialists play in accreditation evaluations? 
(Judges) 

L. What is probably the primary reason accreditation-type evaluations 
have fallen into disfavor? 

(The objectivity and empirical basis of the criteria are questionable, 
iid attention paid to the educational processes not balanced by 
attention to the consequences of the process.) 
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F. What are some extrinsic criteria that could be used in the evaluation 
of vocational education programs? 

(Placement rate, salaries, need for retraining, happirecs, lack of 
"job jumping . ") 

G. What qualifications would an educational evaluator have to have in 
order to justifiably pass judgment on vocational education programs? 

(This is a rhetorical question, but someone is going to have to decide 
sooner or 1 ater . ) 

H. What might some other costs be besides the obvious financial costs? 

(Attitudes, morale, loss of other services to society while students 
are in an education program, excess time needed for the program, etc.) 

I. Is it possible for evaluation-related activities to take place at any 
stage of a program under the CSE Model? 

(Yes) 

J. Are needs assessment and program planning phases in which eva- .^ators 
are normal ly used? 

(Most theorists would answer no, but Alkin et, al_. justify these 
activities on the basis of their value in facilitating decision- 
making. ) 

K. How often are evaluators included in the total curriculum development 
process? Is it a good idea to include evaluators from the moment the 
idea for a project or program is conceived? When are evaluators 
typically brought in on projects or programs? Why? 

(Evaluators are not included in the total process very often. Yes. 
it is a good idea to include them; they may point out potential prob- 
lems in validation, measurement, etc. Evaluators are typically brought 
in at the conclusion of the project or program.) 
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PART IV 
STUDENT SELF-CHECK 



GOAL 14.1 

1. List the four major types of educational evaluation, and state the 
purpose of each type. (14.11) 

2. Proponents of each of the four types of evaluation have stated what 
they believe the evaluator's role should be when conducting that 
particular type of evaluation. Name one person or agency that advo- 
cates each type of evaluation, and state what that person or agency 
believes the role of the evaluator should be. (14.12) 

3. List the operational steps required in conducting each type of 
educational evaluation. Where more than one "model" exists for a 
given evaluation type, list the operational steps of the one that 
you prefer. If a particular model consists of a number of steps 
repeated during successive stages, list the steps only once, but 
include all of them. (14.13) 

a. Goal-attainment: 

b. Judgmental -i ntrinsic : 

c . Judgmental -extrinsic : 

d . Deci s i on-f aci 1 i tati on : 

4. For each of the four types of educational evaluation, list everal 
criteria that it would use in evaluating vocational education. (14. 

a. Goal-attainment 

b. Judgmental-intrinsic criteria 

c. Judgmental-extrinsic criteria 

d. Decision-faci li tation 
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Indicate in the space provided the type of evaluation that is depicted 
in each of the lettered sections below. (14J5) 



The career advisors at two inner-city schools feel that their 
students would profit from exposure to the various occupations 
found in several large corporations located near the schools. 
They contact the companies for the purpose of arranging weekly 
"Orientation-to-Careers" classes on the company sites. The classes 
are to be conducted by people from the companies who are actually 
engaged in the occupations that they will demonstrate and talk 
about. 

By being exposed to the various careers (which are largely semi- 
professional and professional) and the personal and financial 
rewards that they offer, the career advisors feel that the inner- 
city students will be sufficiently motivated to finish high school, 
obtain vocational education, or go on to college. 

The career advisors transform their reasons for the program into 
behavioral objectives and are funded by the district to conduct 
the program for two years, after which the program will be evalu- 
ated on the basis of how many of the students actually finishec 
high school, obtained vocational education, or went on to college. 
The results of the evaluation will be used to determine whether to 
continue or discontinue the program. 



Three years later, after the program in 5 (a) had been dropped 
for failure to meet the objectives, the same two career advisors, 
who still felt that the program was a good idea, decided to try it 
again. After being ridiculed by their peers for even atteinpting 
such a ridiculous project the first time, the two career advisors 
were understandably a little more cautious the second time. They 
began the project by enlisting the aid of the district vocational 
education curriculum specialist (who had completed three evalu- 
ation modules in her degr^JG program) and developing measurable 
behavioral objectives for the program. They then arranged to 
select randomly the students who were to participate in the pro- 
gram, and they expanded the clientele to include a randomly 
selected group of s^tudents from a middle-class suburban high 
school. They also established two control groups who were not to 
receive the benefits of the program: one from the inner-city 
schools and one from the middle-class suburban school. The career 
advisors and the curriculum specialist turned evaluator then 
received a tentative three-year funding grant from the district. 
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Before beginning instruction, the career advisors, the industry 
teachers, and a community advisory group decided on the qualities 
that the students should possess after completion of the program 
and wrote instructional objectives reflecting those qualities; 
they also developed course outlines and instructional materials. 
The materials were field tested during the summer session 
guidance classes at the schools and modified for use when the 
program began in the fall. 

Once the program began, the evaluator visited the instruction 
site and periodically tested the students to appraise their 
progress toward the stated objectives. After the first semester, 
the evaluator reported her observations to the career advisors, 
and several changes were made in the succeeding semesters. At 
the end of two years, the students in the control groups had 
shown little progress, while those in the instructional program 
were either working, involved in vocational education programs, 
or going to college. Because the program was such a success, 
the two career advisors were promoted to administrative positions 
in the district headquarters. 



Baltus Community College, a 2-year college, has had an associate 
degree program in nursing for ten years- Although the graduates 
of the program have experienced no difficulty in obtaining nursing 
licenses, getting transfer credit when they've gone on to the 
state university to obtain their bachelor's degrees has bean 
next to impossible. The state university (and other schools) 
claims that it has no way of knowing how good the program at 
Baltus Community College is. (It doesn't matter that no Baltus 
graduate has ever failed to obtain a state R.N. license.) 
Recognizing the plight of their graduates, the administration at 
Baltus calls in the National League for Nursing, Inc. to validate 
the quality of their program, facilities, and staff. 



A group of recent graduates of an educational evaluation program 
at an unnamed university decide to form their own evaluation con- 
sulting firm. Impressed with their own qualifications and range 
of expertise, the group forms a corporation, and decides to eval- 
uate only vocational programs because three of the five members 
were once vocational teachers. The group unanimously agrees that 
vocational programs should receive continued funding only if at 
least 90% of the graduates are placed in the occupation for which 
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they were trained. The group further believes that vocational 
program graduates should be placed at least one or : /o steps 
up on the career ladder in the occupation for v;ricfi they were 
trained, and that the graduates should be happy the job and 
the employers should be happy with the quality of Hip graduates. 
If a vocational program can meet the above standards, the group 
feels that it could be deemed "successful" and be recommended 
for continued funding. 

With their credo established, the new evaluation group sets out 
to bid on their first evaluation jcb. 
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Appendices 



PART V 
APPENDICES 

Appendix A: 

Possible Study Activity Responses 
GOAL 14.1 



1. Yes, it is compatible with both. 

2. a. Course improvement 

b. Decisions about individuals 

c. Administrative regulation 

3. Because the last two were generally local problems, but course im- 
provement, he felt, could be broadly generalized and would be more 
useful . 

4. At the formative stage because the program and the process would 
still be fluid, or changeable, and the developers v;ould be more re- 
ceptive to change. 

5. An answer that discusses Cronbach's belief that a formal evaluation 
study should be designed to determine the post-course performance of 
well-described groups with respect to defined objectives and side 
effects will be treating the point. 

6. He felt they had minimal effect because they were so far removed in 
time. The results couldn't be used to effect immediate improvement. 
However, he didn't feel that follow-up studies should be abandoned 
because they were useful in measuring attitudes, etc. 



7. a. Determine effectiveness of the institution 

b. Validate hypotheses upon which the sch'iiJ operates 

c. Provide information for guidance of i:idi vidi'r^l students 

d. Reassure staff, Students, and parent^: 1 -i^:? sxhool is working 

e. Provide a sound basis for public relr^'iooi 

f. Help clarify the purposes of the schuol 

8« Behavioral objectives 

9. The changes in behavior of the students. 



10. The development of an instrument or instruments to measure the degree 
to which the objectives of the school /program were met. 

11. Intrinsic criteria are those inherent in the process, such as staff 
qualifications, facilities, etc. Extrinsic criteria are those con- 
cerned with the effects of the process, such as employabi 1 i ty , pro- 
ficiency, etc. 

12. No. Information is not compiled for decision-making but for pro- 
fessional judgment purposes only. Improvement considerations do 
not enter the pi-cture, except in some temporary accreditation situ- 
ations, and even then, thif criteria are still intrinsic in nature, 
and not generally concerned with the effects of the process. 

13. Accreditation. Professional colleagr^s, or "experts" employed by 
the accrediting agency. 

14. Criteria should include such things as staff qualifications, facil- 
ities, equipment, library, etc. 



15. a. Description 
b. Judgment 

16. a. Antecedent 

b. Transaction 

c. Outcomes 

17. It serves as the basis for evaluating the intents of a program. 



18. a. Relati ve--other programs 

b. Absoiute--opinions or personal judgment 

19. No. Exports make the judgment. The evaluator's role is in process- 
ing judgments . 

20. The criteria listed by the students should include those external to 
the program, including, but not limited to placement, job satisfaction, 
employer satisfaction, expert opinion, personal judgment, etc. 



21. a. Pre-formati ve evaluation 

( 1 ) Needs assessment 

(2) Program planning 

b. Formative evaluation 

(1) Implementation evaluation 

(2) Progress evaluation 

c. Summative evaluation 

(1) Document evaluation 

(2) Outcome evaluation 
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22. a(l) Perceived importance of relevant goal area, current status, 

priori ties 

a(2) Information about competing programs, planning documents, etc. 
b(l) Information about the extent to which the program has been 
impl emented 

b(2) Extent to which students are progressing toward the goals 

c(l) Extent to which the program was actually implemented 

c(2) Measures of the results or the outcomes obtained by the program 

23. a. Ascertaining decision areas 

b. Selecting appropriate information 

c. Collecting and analyzing data 

d. Reporting summary information to decision-makers 

24. Alkin feels the judgment role must be discussed and decided upon 
before data collection. If the information collected provides all 
the necessary data for judging, then the evaluation as pure infor- 
mation will speak for itself. 

25. Formative evaluation is for improvement; it involves evaluator 
intervention in the planning, implementation, and process of a pro- 
ject. 

Summative evaluation is done to assess the impact of a program; it 
involves as little evaluator intervention as possible. 

26. Criteria listed here might include goal, objectives, expert opinion, 
test results, costs, social value, etc. 

27. Judgmental -intrinsic criteria 

28. Decision-facil i tation 

29. Goal -attainment 

30. Judgmental -extrinsic criteria 

31 . Goal -attainment 

32. Deci sion-faci 1 i tation 



EKLC 
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Appendix B: 

Possible Self-Check Responses 



GOAL 14.1 

1. List the four major types of educational evaluation, and state the 
purpose of each type. (14.11) 

a. Goal-attainment-purpose: determining if goals w^re met 

b. Judgmental -intrinsic criteria-purpose: valuing 
a program based on intrinsic criteria. 

c. Judgmental-extrinsic criteria-purpose: valuing the effects of 
a program based on extrinsic criteria. 

d. Decision-facilitation -purpose: providing a service to deci- 
sion-makers for program improvement. 



2. Proponents of each of the four types of evaluation have stated what 
they believe the evaluator's role should be when conducting that par- 
ticular type of evaluation. Name one person or agency that advo- 
cates each type ui evaluation, and state what that person or agency 
believes the role of the evaluator should be. (14.12) 



a. Goal-attainment 



b. Judgmental -intrinsic 

c. Judgmental -extrinsic 

d. Decision-fac-^1 i tat ion 



Ralph Tyler 
Robert Hammond 
Metfessel & Michael 

Various agencies 



Michael Scriven 
Robert Stake 

Mai com Provus 

D. Stufflebeam 

Alkin or CSE 



Appraising goal attainment 
Appraising goal attainment 
Appraisal and recommendation 

Observe, check, judge 

Observe and judge 
Aid, observe, judge 

Aid, observe, report, 

recommend 
Aid, observe, record, 

recommend 
Aid, observe, report, 

recommend 
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List the operational steps required in conducting each type of 
educational evaluation. Where more than one "model" exists for a 
given evaluation type, list the operational steps of the one that 
you prefer. If a particular model consists of a number of steps 
repeated during successive stages, list the steps only once, but 
include all of them. (14.13) 

a. Goal -attai nment: 

Tyler-(l) formulate goals, (2) transform goals into behavioral 
objectives, (3) conduct program, (4) measure students to 
assess the degree to which objectives were met. 

Hammond-(l) isolate aspects to be evaluated, (2) define variables, 
(3) specify objectives, (4) assess the behavior described 
in the goals, (5) analyze the results. 

Metfessel and Michael-(l) involve members of the total community, 
(2) construct broad goals and specific objectives, (3) tran- 
slate objectives into communicable and usable form, (4) de- 
velop measurement instruments, (5) conduct periodic measure- 
ments, (6) analyze data from measurements, (7) interpret 
analyzed data, (8) formulate recommendations for change or 
modificiation. 



b. Judgmental~i ntri nsi c 

(1 ) establ ish standards 

(2) conduct self-study (measure performance against standards) 

(3) observation by evaluation team (on-site visitation) 

(4) judgment by evaluation team and publication of results 

(5) periodic reevaluation 



c . Judgmental -extrinsi c: 

Stake-(l) determine anteced?^nts , transactions, and outcomes 

for each of the two phases, (2) describe and judge. In 
the description phase, intents are compared with actual 
observations. In the judgment phase, the evaluator de- 
termines standcirds (relative and absolute), and makes 
judgments based on them. 

Scriven-(l) Initially formulate goals that will be regularly 

reexamined and modified, (2) develop a pool of test items 
that are operational versions of the goals, (3) obtain ex- 
ternal judgments of the goals and the test item pool, in 
order to validate consistencies, detect omissions and 
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discrepancies between implicit and stated goals. (4) A 
successful match is essential between goals and course 
content, goals and examination content, and course and 
examination content. (5) Comparing outcomes for programs 
versus control groups-can include comparisons on dimen- 
sions of knowledge, comprehension, attitudes and values, 
perceptual-motor skills. (6) Other external curriculum 
effects to examine are those on the teacher, his or her 
colleagues, other students, administrators, parents, the 
institution and the taxpayer. 



d. Decision-facil i tation: 

Provus-Five stages to the model, with the same three steps con- 
ducted at each stage: (1) define program standards, 
(2) determine if discrepancies exist between standards 
and performance, and (3) use discrepancy ir ' -mation to 
change performance or program standards. Ihdse three 
steps are followed in each of the five stages: design, 
installation, process, product, and oronram comnanson. 

Stufflebeam-{1 ) delineate information requirements, (2) obtain 
information to meet those requirements, (3) provide the 
information to decision-makers. These three steps are 
followed in the four phases of the CIPP evaluation model: 
context, input, process and product. 

Alkin or CSE-(l) ascertain the decision areas of concern, 

(2) select appropriate 'information, (3) collect and ana- 
lyze data, (4) report summary information to decision- 
makers. These four steps are followed in the six stages 
of CSE decision-facilitation evaluation: needs assess- 
ment, program planning evaluation, implementation evalua- 
tion, progress evaluation, documentation evaluation, and 
outcome evaluation. 



For each of the four types of educational evaluation, list several 
criteria that it would use in evaluating vocational education. (14 

a. Goal-attainment ^ 

student progress toward objectives; job placement (if that s 
an objective) 

b. Judgmental-intrinsic criteria 

years of occupational experience of faculty; ouality of 
tools and equipment; type of instruction (methodology); 
number and type of books in library relating to vocations; 
physical facilities; type of programs offered; financial 
support; etc. 
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c. Judgmental -extrinsic crHeria 

effects of the program on learners, both planned 

and unplanned; the effect of the program on the 

occupations for wfiich training is offered; the 

"quality" of the graduates relative to graduates 

of other programs; how well students learned in 

the program; relative quality of educational goals; etc. 

d. Decision-faci 1 i tat ion 

This answer could include nearly any criteria that 
a decision-maker needed information on in order to 
improve (or even maintain) an educational program. 



Indicate in the space provided the type of evaluation that is de- 
picted in each of the lettered sections below. (14.15) 

a. Goal-attainment 

The career advisors at two inner-city schools feel that 
their students would profit from exposure to the various 
occupations found in several large corporations located near 
the schools. They contact the companies for the purpose of 
arranging weekly "Orientation-to-Careers" classes on the 
company sites. The classes are to be conducted by people 
from the companies who are actually engaged in the occupa- 
tions that they will demonstrate and talk about. 

By being exposed to the various careers (which are largely 
semi-professional and professional), and the personal and 
financial rewards these offer, the career advisors feel 
that the inner-city students will be sufficiently motivated 
to finish high school, obtain vocational education, or go 
on to college. 

The career advisors transform their reasons for the program 
into behavioral objectives and are funded by the district 
to conduct the program for two years, after which the program 
will be evaluated on the basis of how many of the students 
actually finished high school, obtained vocational education, 
or went on to college. TheVesults of the evaluation will be 
used to determine whether to continue or discontinue the 
program. 



b. Decision-faci 1 i tation 

Three years later, after the program in 5 (a) har been 
dropped for failure to meet the objectives, the same two 
career advisors, who still felt that the p.^ogram was a good 
idea, decided to try it again. After being ridiculed by 
their peers for even attempting such a ridiculous project the 
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first time, the two career advisors were understandably a 
little more cautious the second time. They began the project 
by enlisting the aid of the district vocational education 
curricjlum specialist (who had compelted three evaluation 
module, in her degree program) and developing measurable behavioral 
objectives for the program. They than arranged to select randomly 
the students who were to participate in the program, and they ex- 
panded the clientele to include a randomly selected group of stu- 
dents from a middle-class suburban high school. They also estab- 
lished two control groups who were not to receive the benefits of 
the program: one from the inner-city schools and one frorr the 
middle-class suburban school. The career advisors and the curric- 
ulum specialist turned evaluator then received a tentative three- 
year funding grant from the district. 

Before beginning instruction, the career advisors, the indus- 
try teachers, and a community advisory qroup decided on the 
qualities that the students shCv,''d possess after completion of 
the program and wrote instructional objectives reflecting those 
qualities; they also develop-^d course outlines and instructional 
materials. The materials were /ield tested during the summer 
session guidance classes at th^* schools and modified for use 
when the program began in the rail. 

Once the program began, the evaluator visited the instruction 
site and periodically tested the students to appraise their 
progress toward the stated objectives. After the first semes- 
ter, the evaluator reported her observations to the career 
advisors, and several changes were made in the succeeding se- 
mesters. At the end of two years, the students in the control 
groups had shown little progess, while those in the instruc- 
tional program were either working, involved in vocational 
education programs, or going to college. Because the program 
was such a success, the two career advisors were promoted to 
administrative positions in the district headquarters. 



Judgmental-intrinsic criteria 

Baltus Community College, a 2-year college, has had an asso- 
ciate degree program in nursing for ten years. Although the 
graduates of the program have experienced no difficulty in ob- 
taing nursing licenses, getting transfer credit when they've 
gone on to the state university to obtain their bachelor's 
degrees has been next to impossible. The state university (and 
other schools) claims that it has no way of knowing. how good 
the program at Baltus Community College is. (It doesn't matter 
that no Baltus graduate has ever failed to obtain a state R.N. 
license.) Recognizing the plight of their graduates, the ad- 
ministration at Baltus calls in the National League for Nursing, 
Inc. to validate the quality of their program, facilities, a« d 
staff. 
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Judgmental-extrinsic criteria 



A group of recent graduates of an educational evaluation 
program at an unnamed university decide to form their own 
evaluation consulting firm. Impressed with their own quali- 
fications and range of expertise, the group forms a corpora- 
tion, and decides to evaluate only vocational programs be- 
cause three of the five members were once vocational teachers. 
The group unanimously agrees that vocational programs should 
receive continued funding only if at least 90% of the grad- 
uates are placed in the occupation for which they were trained. 
The group further believes that vocational program graduates 
should be placed at least one or two steps up on the career 
ladder in the occupation for which they were trained, and that 
the graduates should be happy on the job and the employers 
shoulH be happy with the quality of the graduates. If a voca- 
tio orogram can meet the above standards, the group feels 
tha-. could be deemed "successful" and be recommended for 
continued funding. 

With their credo established, the new evaluation group sets 
out to bid on their first evaluation job. ^ 



57 



*OPO 69 2. 198-1 97# 



-46- 



